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Abstract 

In  today’s  budget-constrained  environment  with  aging  equipment  and  rapid 
technology  turnover,  the  commander’s  requirement  to  prioritize  and  justify  funding 
requests  has  essentially  become  an  exercise  in  mission  assurance.  This  study  addresses 
the  Theater  Ballistic  Missile  Defense  Systems  (TBMDS)  mission  assurance  problem 
generated  from  the  complexity  of  the  political,  organizational  and  physical/architectural 
environments. 

The  Mission-Oriented  Success  Analysis  and  Improvement  Criteria  (MOSAIC) 
management  approach  provides  commanders  a  framework  for  mission  assurance  of 
TBMDS.  This  study  specifically  incorporates  a  discrete  event  simulation  analysis  and  the 
Mission  Assurance  Analysis  Protocol  (MAAP)  to  identify  the  most  effective 
improvement  areas  under  the  MOSAIC  approach. 

At  a  high  level  of  abstraction,  analysis  from  a  discrete  event  simulation  of  the 
TBMDS  prioritized  the  component  reliability  from  the  most  to  least  impactful  as 
shooter/interceptor,  C2  facility,  Radar  and  Communication.  The  MAAP  expanded 
success  profiles  simultaneously  address  multiple  environmental  factors,  including 
reliability,  for  each  key  objective  needed  for  the  mission.  At  an  equally  high  level  of 
abstraction,  these  profiles  identified  that  the  key  objective  to  strengthen  US  security 
relations  of  the  TBMDS  can  be  achieved  successfully  and  three  key  objectives,  negate 
effectiveness  of  Theater  Ballistic  Missile  (TBM),  negate  likelihood  of  TBM,  and 
protect/defend  forces/population  centers,  had  significant  mission  assurance  gaps.  With 
improvement  areas  and  mission  assurance  gaps  identified,  commanders  are  equipped 
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with  the  required  information  for  funding  requests  and  confidence  in  the  success  of  their 
mission. 
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MISSION  ASSURANCE  ANALYSIS  OF  THEATER  BALLISTIC  MISSILE 

DEFENSE  SYSTEMS 

Chapter  1.  Introduction 

1.1.  Background 

Commanders  in  today’s  budget  constrained  US  military  have  the  opportunity  to 
improve  mission  enabling  processes  and  systems  within  their  command  by  eliminating 
unnecessary  redundancies  and  non- value  added  processes  (Chavez,  2011).  At  the  same 
time,  the  available  military  budget  is  dispersed  under  a  prioritized  requirement  driven 
process  (ACC,  2011).  The  commander’s  decision  to  improve,  remove,  or  add  a 
process/system  essentially  becomes  an  exercise  in  mission  assurance.  In  addition, 
technology  is  continually  changing  the  battlespace  to  support  the  warfighter  in  their 
escalating-capabilities  race  for  superiority.  Therefore,  commanders  at  all  levels  have  an 
ever  present  need  to  efficiently  maintain  mission  assurance  of  their  enabling  processes 
and  systems  in  the  midst  of  this  dynamic  environment. 

Historically,  the  major  portion  of  any  mission  assurance  program  was  risk 
management.  Traditional  risk  management  approaches  attempted  to  foresee  and  mitigate 
issues  that  could  hinder  mission  success  with  infinite  levels  of  detail.  “The  traditional  risk 
management  process  is  the  sequential  application  of  five  discrete  action  steps:  identify, 
measure,  select,  implement  and  monitor”  (Burlando,  1990).  Not  only  was  this  process 
time  consuming,  but  was  also  typically  accomplished  by  a  few  isolated  technicians. 
Commanders  today  cannot  afford  to  manage  mission  success  in  such  a  fashion.  “Mission 
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success  in  complex  settings  demands  a  collaborative  management  approach  that 
effectively  coordinates  task  execution  and  decision-making  activities  among  all 
participating  groups”  (Alberts  et.  ah,  2007). 

1.2.  Motivation 

In  2010,  US  European  Command  desired  to  better  understand  the  limitations 
involved  with  completing  a  war  plan  that  included  system  of  systems  surrounded  by 
dynamic  political,  organizational  and  physical/architectural  environments.  This  study  is  a 
part  of  a  larger  Air  Force  Institute  of  Technology  (AFIT)  effort  and  uses  a  Theater 
Ballistic  Missile  Defense  System  (TBMDS)  case  study  to  propose  a  mission  assurance 
process  able  to  handle  the  complexity  of  a  system  of  systems  and  explore  the  possibility 
that  a  commander  can  manage  system  resources/processes  using  mission  assurance  in 
pursuit  of  operational  mission  success. 

Though  research  on  mission  assurance  is  not  new,  previous  research  has  mainly 
focused  on  program/project/process  mission  assurance  or  cyberspace  mission  assurance. 
Many  of  the  approaches  developed  within  these  areas  leverage  traditional  risk 
management  in  establishing  mission  assurance.  However,  with  all  the  mission  assurance 
tenets  of  reliability,  quality  assurance,  force  protection,  system  safety,  etc.,  the  central 
focus  of  mission  assurance  should  be  mission  success  (Bryant,  2011;  DoDb,  2010)  as 
opposed  to  hazard  risk  aversion. 

The  inherent  joint  and  coalition  operational  complexities  of  a  TBMDS  make  it  a 
great  research  candidate  for  developing  a  mission  assurance  approach.  Additionally,  in 
the  current  round  of  downsizing  and  budget  cuts,  commanders  are  even  more  reliant  on 
mission  assurance  to  make  sound  budget  decisions  as  the  obvious  mission  areas  have 
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already  been  optimized.  In  the  midst  of  all  the  politics  and  missile  defense  restructuring 
plans,  military  operations  are  still  being  conducted,  and  commanders  at  all  levels  still 
require  mission  assurance  of  the  systems  they  employ  from  day  to  day.  Under  the  current 
Administration’s  strategy,  the  United  States  is  pursuing  “a  phased  adaptive  approach  to 
missile  defense  within  each  region  that  is  tailored  to  the  threats  and  circumstances  unique 
to  that  region”  (DoDa,  2010).  Therefore,  mission  assurance  of  a  region’s  Theater  Ballistic 
Missile  Defense  (TBMD)  will  necessarily  rely  on  the  input  of  the  region’s  commanders. 
Furthermore,  due  to  the  nature  of  DoD  acquisitions,  systems  and  the  communication 
networks  they  use  are  generally  designed/managed  separately.  This  often  makes  it 
difficult  for  commanders  to  prioritize  funding  requirements  for  a  capability  when  trying 
to  improve,  maintain,  or  streamline  a  particular  mission  area. 

1.3.  Research  Objectives  and  Structure 

From  the  TBMDS  perspective,  prior  research  has  focused  mainly  on  the 
architecting  of  system  coverage  designs  versus  providing  commanders’  mission 
assurance  of  their  TBMD  systems.  This  research  effort  attempts  to  look  at  U.S.  European 
Command’s  (USEUCOM)  TBMD  system  to  help  the  commanders  identify  risk  and 
opportunity  that  can  have  an  impact  on  the  ability  to  achieve  the  Department  of  Defense’s 
(DoD’s)  objective  to  protect  forward-deployed  and  expeditionary  elements  of  the  U.S. 
armed  forces  as  well  as  friends  and  allies  of  the  United  States  from  a  given  region’s 
ballistic  missile  threats  (Alberts,  et  al.,  2008;  SECDEF,  1995). 
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This  study  was  conducted  using  the  Software  Engineering  Institute  (SEI)  Mission- 
Oriented  Success  Analysis  and  Improvement  Criteria  (MOSAIC)1  approach  to  establish 
confidence  that  the  key  objectives  of  the  TBMDS  are  achieved  successfully  or  area(s)  are 
identified  that  require  attention  in  order  to  attain  that  confidence/  provide  mission 
assurance  (Alberts  et.  ah,  2007).  The  MOSAIC  approach  employed  two  separate  but 
supporting  assessments.  The  first  assessment  was  a  discrete  event  simulation  analysis 
using  Rockwell’s  ARENA  simulation  tool.  The  ARENA  simulation  model  helped 
prioritize  improvement  areas  of  a  TBMDS  by  looking  at  the  system  components  to 
determine  which  area(s)  or  component(s)  would  contribute  most  to  increasing  mission 
success  (Arena,  2007).  This  tool  provided  statistically  significant  data  and  quantified 
improvement  projections  to  support  a  TBMDS  prioritized  funding  request.  The  second 
assessment  of  this  study  used  the  Mission  Assurance  Analysis  Protocol  (MAAP)2  to 
simultaneously  address  multiple  environmental  factors,  including  reliability,  for  each  key 
objective  needed  for  mission  assurance  (Alberts  et.al.,  2008). 

With  improvement  areas  and  mission  assurance  gaps  identified,  commanders  are 
provided  with  the  required  information  for  funding  requests  and  confidence  in  the  success 
of  their  mission. 

1.4.  Research  Scope 

This  effort  will  leverage  representative  reliability  data  to  demonstrate  how 
mission  assurance  for  a  TBMDS  could  be  assessed.  The  MAAP  that  will  be  executed  is 

1  MOSAIC  was  developed  by  the  Software  Engineering  Institute  (SEI)  at  Carnegie  Mellon  University, 

Copyright  2007.  Details  on  MOSAIC  are  provided  in  Chapter  2. 

2 

MAAP  was  developed  by  the  Software  Engineering  Institute  (SEI)  at  Carnegie  Mellon  University, 
Copyright  2008.  Details  on  MAAP  are  provided  in  Chapter  2. 
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also  illustrative  as  a  full  MAAP  assessment  with  extensive  manpower,  full  support  from 
process  owners  and  enterprise-wide  alignment  of  key  objectives  are  beyond  the  scope  of 
this  research  effort.  As  such,  only  select  top-level  key  objectives  were  included  to 
demonstrate  the  MAAP  assessment  flow  and  capabilities. 
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Chapter  2.  Literature  Review 

The  larger  body  of  previous  research  has  focused  on  the  architecture  of  the 
TBMDS  which  is  essential  to  understand  before  tackling  any  analysis  of  this  system  and 
the  concept  of  mission  assurance.  Even  though  many  of  the  specific  mission  assurance 
efforts  do  not  adequately  address  system  complexity  or  are  not  operationally  relevant, 
they  are  important  to  recognizing  the  necessary  diversity  of  mission  assurance 
approaches.  This  chapter  discusses  TBMDS  and  a  commander’s  inherent  need  for 
mission  assurance.  It  then  addresses  the  challenges  commanders  face  trying  to  achieve 
mission  assurance  and  the  current  areas  of  mission  assurance  research.  After  a  thorough 
review  of  mission  assurance,  the  chapter  will  conclude  with  a  mission  assurance 
approach  for  TBMDS. 

2.1.  Theater  Missile  Defense  Program  Concepts  and  Current  State 

The  Theater  Missile  Defense  (TMD)  programs  can  be  considered  a  system  of 
systems  or  family  of  systems  program.  Multiple  sources  speak  to  these  programs  and 
their  composition.  Army  Field  Manual  (FM)  3-01  describes  a  typical  TMD  architecture  as 
a  combination  of  shared  and  dedicated  components  that  consist  of  long  range  sensor(s), 
an  In-Flight  Interceptor  Communications  System  and  a  battle  management,  command 
and  control  and  communications  (C3)  network  (AFM  3-01,  2009).  The  long  range 
sensor(s)  provide  early  warning  and  surveillance  of  ballistic  missile  launch  while  the  In- 
Flight  Interceptor  destroys  the  missile  in  flight  (from  boost  phase  to  descent  phase  by 
offensive  or  defensive  means).  The  battle  management  C  network  supports  sensor 
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management,  data  processing  and  dissemination,  and  command,  control  and 
communications . 

The  TMD  programs  directly  support  many  separate  programs  efficiently  by 
accomplishing  tasks  that  otherwise  would  have  to  be  achieved  by  separate  programs.  The 
Command  and  Control,  Battle  Management  Communications  (C2BMC)  provides 
interoperability  that  is  essential  for  joint  TMD  operations.  The  Missile  Defense  Agency 
(MDA)  purposely  established  an  architecture  that  all  the  Services  can  build  upon  that 
includes  improving  early  warning  and  dissemination,  ensuring  communications 
interoperability,  and  upgrading  command  and  control  centers.  These  capabilities  are 
absolutely  critical  to  the  success  of  the  overall  U.S.  Missile  Defense  system.  They  are 
often  referred  to  as  the  glue  that  holds  the  architecture  together  and  ensures  that  the 
whole  is  greater  than  the  sum  of  its  parts  (JTMP,  2004). 
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Figure  1  depicts  the  basic  engagement  sequence  without  the  separate  C2BMC 
interaction  and  a  futurist  launch  on  remote  concept.  The  current  TBMDS  response  is 
initiated  when  remote  sensor(s)  detect  a  ballistic  missile  launch  as  shown  as  step  number 
1  in  Figure  1.  Remote  sensor(s)  relay  surveillance  and  track  information  to  the  C2BMC 
which  tracks  the  ballistic  missile  threat  with  the  theater’s  own  remote  sensor.  The 
tracking  and  surveillance  by  these  remote  sensors  is  depicted  by  step  2  in  Figure  1.  The 
C2BMC  determines  the  intent  of  the  missile,  identifies  a  required  course  of  action,  and 
sends  an  engagement  order  to  the  launch  system  for  hostile  missile  threats.  Current 
TBMDS  require  that  the  launch  system’s  own  radar  be  able  to  identify  the  threat  missile 
before  launch  therefore  reversing  the  order  of  steps  3  and  4  in  Figure  1  (DoDa,  2010). 
Finally,  in  a  successful  mission  scenario,  the  launch  system  radar  refines  the  trajectory 
and  successfully  intercepts  the  ballistic  missile  as  shown  in  step  5  in  Figure  1. 

The  engagement  sequence  described  above  conveys  the  distinct  functions  of  the 
various  components  of  the  TBMDS.  TBMDS  distinct  functions  have  driven  research  and 
development  to  be  very  component  specific.  As  such  a  system  overview  will  briefly 
outline  the  individual  components  beginning  with  the  C2BMC  also  referred  to  as  BMC3, 
BMC3I,  BM/C3  and  BM/C3I  over  the  years.  The  C2BMC  program  has  the  responsibility 
to  integrate  the  other  BMDS  components.  The  C2BMC  in  and  of  itself  is  a 
conglomeration  of  capabilities.  The  command  and  control  element  covers  the 
functionalities  that  support  planning  and  situational  awareness.  The  battle  management 
element  is  responsible  for  supporting  the  combatant  commander  in  making  the  decision 
to  commit  a  weapon  system,  while  the  communications  element  is  responsible  for 
providing  the  communications  capabilities  for  the  BMDS  (Miller,  2008).  The  C2BMC 
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strives  to  “enable  coordinated,  real-time  decision-making  by  war-fighters  and  leaders 
across  the  globe,  up  to  and  including  the  secretary  of  defense  and  the  president  of  the 
United  States”  (Jay,  2009).  The  C2BMC’s  specific  mission  is  to  “provide  a  combatant 
command  decision  aid  to  integrate  and  globally  synchronize  missile  defense  systems  and 
operations.”  To  fulfill  this  mission,  the  C2BMC  provides  five  main  functions:  1)  the 
communications  links  and  connectivity  between  ballistic  missile  defense  components,  2) 
a  battle  management  function  to  allow  a  shoot/look/shoot  approach  that  maximizes 
BMDS  effectiveness,  3)  control  of  the  BMDS  radars,  4)  real-time  awareness  of  the 
battlespace  to  include  interoperability  with  NATO,  and  5)  advanced  battle  planning 
capability  that  enables  warfighters  to  place  BMDS  assets  in  optimal  locations  in 
anticipation  of  an  upcoming  battle  (Jay,  2009). 

On  September  17,  2009  President  Obama  announced  the  U.S.  decision  to  adopt  a 
new  approach  to  ballistic  missile  defense  in  Europe  called  the  European  Phased  Adaptive 
Approach  (EPAA).  The  EPAA  calls  for  “operating  BMD-capable  Aegis  ships  in 
European  waters  to  defend  Europe  from  potential  ballistic  missile  attacks  from  countries 
such  as  Iran”  (O’Rourke,  2010).  In  support  of  the  EPAA,  USEUCOM’s  C2BMC 
capabilities  were  recently  improved  by  the  “activation  of  C2BMC  servers,  providing 
track  forwarding  to  in-theater  BMD  Aegis  ships,  situational  awareness  of  the  regional 
BMD  mission,  and  control  for  the  forward-based  AN/TPY-2  radar  in  support  of  the 
Defense  of  Israel”  (MDA,  2011).  In  addition,  in  March  2011,  the  United  States 
announced  the  deployment  of  the  US S  Monterey  to  the  Mediterranean  to  begin  a 
sustained  deployment  of  Aegis  BMD-capable  ships  in  support  of  the  EPAA  (EPAA  and 
NATO  Missile  Defense,  May  201 1).  The  Aegis  BMD-capable  ships  provide  interceptor 
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capability  in  the  European  theater  with  SM-2  and  SM-3  missiles  (O’Rourke,  2010). 

The  last  TBMDS  component,  the  long  range  sensor(s),  have  seen  significant 
improvement  since  the  inception  of  the  AN/TPY-2  X-band  radars  providing  detection 
and  tracking  of  ballistic  missiles  at  very  long  ranges.  The  direct  radar-CSBMC  link  adds 
valuable  seconds  to  the  decision  making  process  between  missile  launch  and  interception 
(Graves,  2008  and  94th  AAMDC,  2009).  These  improvements  provide  greater  flexibility 
and  mission  assurance  of  USEUCOM’s  TBMDS. 

2.2.  Concept  of  Mission  Assurance 

DoDD  3020.40  defines  mission  assurance  as: 

A  process  to  ensure  that  assigned  tasks  or  duties  can  be  performed  in 
accordance  with  the  intended  purpose  or  plan.  It  is  a  summation  of  the 
activities  and  measures  taken  to  ensure  that  required  capabilities  and  all 
supporting  infrastructures  are  available  to  the  Department  of  Defense  to 
carry  out  the  National  Military  Strategy.  It  links  numerous  risk 
management  program  activities  and  security-related  functions,  such  as 
force  protection;  antiterrorism;  critical  infrastructure  protection;  IA; 
continuity  of  operations;  chemical,  biological,  radiological,  nuclear,  and 
high  explosive  defense;  readiness;  and  installation  preparedness  to  create 
the  synergy  required  for  the  Department  of  Defense  to  mobilize,  deploy, 
support,  and  sustain  military  operations  throughout  the  continuum  of 
operations.  (DoD,2010) 

Mission  assurance  by  this  definition  is  a  must  for  any  commander  (Abercrombie 

et.  al.,  2010;  Evans,  2010;  Hale,  2010;  Hale,  et.al.,  2010).  However,  mission  assurance 

management  shares  similar  management  considerations  with  risk  management  and 

information  assurance  like,  scope,  definition  flexibility,  rapid  technology  change  rate  and 

subjectivity  (Grimaila  et.  al.,  2010).  First,  scoping/bounding  the  mission  to  the 

appropriate  level  is  an  art  unto  itself.  Scoping  the  mission  too  large  means  the 

commander  does  not  have  the  authority  to  control/influence  all  the  components  required 
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to  fulfill  the  mission.  Scoping  the  mission  too  small,  generally  leads  to  micromanaging  a 
subordinate’s  mission  area  and  neglect  of  other  mission  areas  (NIST,  2010).  Secondly, 
the  term  “mission  assurance”  is  very  flexible.  It  has  and  can  be  used  throughout  the  life 
cycle  of  a  system  (Guarro,  2007)  or  to  ensure  any  objective  is  met.  Air  Force  Doctrine 
Document  3-12  describes  mission  assurance  as  “measures  required  to  accomplish 
essential  objectives  of  missions  in  a  contested  environment”  (Department  of  the  Air 
Force,  2010).  In  this  research  paper  however,  mission  assurance  will  refer  to  the 
operational  part  of  the  system’s  lifecycle  as  it  relates  to  the  mission  objectives.  This  will 
ensure  once  again  that  the  commander  has  the  authority  to  control/influence  the 
components  required  to  fulfill  the  mission.  Thirdly,  with  the  current  rate  of  change  in 
technology,  the  suggested  rapid  refresh  rates  and  interoperability  improvement  initiatives, 
the  components  required  to  fulfill  the  mission  are  changing  constantly.  Thus  analysis 
done  to  support  mission  assurance  could  effectively  be  obsolete  the  moment  it  is 
collected  (Isaacs,  1999).  Finally,  mission  assurance  is  inherently  subjective  (Weaver, 
2005).  Therefore,  mission  assurance  is  dependent  upon  the  individual  in  charge.  Two 
different  individuals  may  have  differing  opinions  on  the  mission  assurance  of  the  exact 
same  system.  These  aspects  of  mission  assurance  and  more,  make  research  on  mission 
assurance  relatively  area/system  specific. 

The  current  areas  of  research  in  mission  assurance  tend  to  fall  in  four  groups 
cyberspace  mission  assurance,  program/project  mission  assurance  which  tends  to  be 
acquisition/business  focused,  NASA’s  Mission  Operational  Assurance  (MO A),  and 
process  mission  assurance.  The  rest  of  this  section  will  briefly  touch  on  these  research 
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areas  and  determine  the  approach  that  fits  the  needs  of  a  TBMDS  commander’s  mission 
assurance  requirements  best. 

Air  Force  Doctrine  Document  3-12,  Cyberspace  Operations,  reflects  that  “mission 
assurance  entails  prioritizing  Mission  Essential  Functions  (MEFs),  mapping  mission 
dependence  on  cyberspace,  identifying  vulnerabilities,  and  mitigating  risk  of  known 
vulnerabilities”  (Department  of  the  Air  Force,  201 1).  In  their  Journal  of  Strategic 
Security  article,  “The  Science  of  Mission  Assurance,”  Jabbour  and  Muccio  use  the  same 
terms  to  describe  a  cyberspace  mission  assurance  methodology:  prioritization,  mission 
mapping,  vulnerability  assessment  and  mitigation  (Jabbour  and  Muccio,  2011).  These 
approaches  have  essentially  leveraged  the  traditional  risk  management  steps  (identify, 
measure,  select,  implement  and  monitor)  on  a  subset  of  system  functions  effectively 
watering  down  mission  assurance  to  merely  avoiding  negative  consequences  to  MEFs 
(Burlando,  1990;  Grimaila  et.  al.,  2010). 

Program/project  mission  assurance  research  addresses  mission  assurance  from  a 
systems  engineering  perspective  and  breaks  mission  assurance  into  six  core  processes; 
requirements  analysis  and  validation,  design  assurance,  manufacturing  assurance, 
integration,  testing,  and  evaluation,  operational  readiness  assurance,  and  mission 
assurance  reviews  and  audits  (Guarro,  2007).  Though  program  mission  assurance  is 
essential  for  the  acquisition  of  major  engineering  endeavors,  analyzing  fielded 
operational  systems  require  a  different  perspective  on  mission  assurance. 

The  third  area  of  current  mission  assurance  research  focused  specifically  on 
NASA’s  Mission  Operations  Assurance  (MO A).  NASA  coined  the  acronym  MOA  to 
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distinguish  between  their  approaches  to  program  mission  assurance  and  assurance  of 
operational  mission  success.  They  laid  out  four  MOA  program  requirements: 

1.  MOA  shall  independently  assess  project  risks  throughout  mission  operations. 

2.  MOA  shall  independently  assess  the  project’s  operational  readiness  to  support 
nominal  and  contingency  mission  scenarios. 

3.  MOA  shall  implement  the  project’s  problem/failure  reporting  system  to  comply 
with  Jet  Propulsion  Laboratory’s  Anomaly  Resolution  Standard. 

4.  MOA  shall  provide  training  on  problem  reporting  for  the  flight  team.  (Bryant, 

2011) 

This  approach  once  again  focuses  on  traditional  risk  management  techniques  to  merely 
avoid  negative  consequences. 

The  last  area  of  research  covered  in  this  chapter  focuses  on  the  Software 
Engineering  Institute  (SEI)  Mission-Oriented  Success  Analysis  and  Improvement  Criteria 
(MOSAIC)  approach  to  process  mission  assurance.  MOSAIC  entails  “a  suite  of 
advanced,  risk-based  analysis  methods  for  assessing  complex,  distributed  programs, 
processes,  and  information-technology  systems”  (Alberts  et.  al.,  2007).  Unlike  traditional 
risk  management  approaches,  MOSAIC  uses  outcome-based  risk  management,  which 
assumes  an  aggregate  view  of  risk  that  forecasts  the  most  likely  outcome  from  a  range  of 
possibilities  and  attempts  to  maintain  the  overall  level  of  risk  within  acceptable  limits. 
With  the  MOSAIC  management  paradigm  shown  in  Figure  2,  commanders  can  establish 
and  maintain  confidence  in  success  at  any  place  in  the  life  cycle  and  help  provide 
assurance  at  the  mission,  system,  and  program  levels. 
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Figure  2.  MOSAIC  Management  Paradigm  (Alberts  et.  al.,  2008) 

This  methodology  prescribes  a  mission-focused  approach,  which  facilitates  consideration 

of  complex  environmental  interactions.  The  SEI  MOSAIC  approach  “initially  focused  on 
managing  success  in  projects,  programs,  and  operational  processes”  (Alberts  et.  al., 

2007).  This  research  paper  will  attempt  to  expand  this  methodology  to  the  mission 
assurance  of  the  TBMD  system  of  systems. 

2.3.  MOSAIC  for  TBMDS 

Review  of  TBMDS  and  mission  assurance  studies,  support  the  use  of  MOSAIC  as 
an  efficient  mission  assurance  process  for  an  operational  system  of  systems  in  dynamic 
and  complex  environments.  MOSAIC  was  developed  as  a  modular  design  to  allow 
flexibility  in  mission  assurance  analysis  to  reflect  the  unique  nature  of  the  organization 
and  environment.  SEI  MOSAIC  is  comprised  of  a  “flexible  set  of  methods  that  can  be 
used  to  solve  a  variety  of  analysis  problems”  (Alberts  et.  al.,  2007).  It  uses  protocols, 
techniques  and  supporting  artifacts  to  describe  the  analysis  methods.  The  protocol  defines 
the  basic  framework  for  conducting  the  analysis  and  lays  out  the  sequence  of  activities  to 
be  preformed.  The  techniques  describe  any  methods  that  can  be  used  to  perform  the 
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activities  and  the  supporting  artifacts  are  any  charts,  templates,  etc.  used  in  the  process. 
Each  SEI  MOSAIC  analysis  method  and  associated  protocols,  techniques  and  supporting 
artifacts  are  tailored  to  the  specific  situation.  Mission  assurance  analysis  for  the  TBMDS 
will  use  the  SEI  Mission  Assurance  Analysis  Protocol  and  a  discrete-event  simulation 
protocol  based  on  the  simulation  study  steps  from  a  Discrete-Event  System  Simulation 
(Banks,  et  al.,  2010). 

2.3.1.  SEIMAAP 

MAAP  expands  risk-analysis  to  include  use  of  an  operational  model  to  determine 
how  potential  events  will  affect  the  current  value  of  the  potential  for  success  and  an 
uncertainty  range  to  represent  the  educated  guesswork  required  in  any  risk-analysis.  The 
assistance  of  the  operation  model  provides  commanders  a  basis  for  event  analysis  and 
consequence  generation,  while  the  uncertainty  range  represents  a  stochastic  nature  of 
reality.  In  addition,  MAAP  includes  a  sensitivity  analysis  for  an  event’s  effect  on  key 
process  objectives.  Since  mission  assurance  is  based  on  the  commander’s  ability  to  have 
confidence  in  the  process,  these  expansions  become  a  multi-dimensional  risk  analysis  in 
which  the  commander  can  have  confidence  (Alberts  et.al.,  2008). 

MAAP  is  a  three  phase  process:  Phase  1,  Preparing  for  the  Assessment,  Phase  2, 
Conducting  the  Assessment  and  Phase  3,  Post  Assessment  Activities.  However,  the  core 
assessment  activities  are  preformed  in  phase  2,  which  typically  involves  a  team  of  experts 
developing  a  detailed,  descriptive  operational  model  of  the  process  being  assessed. 
Additional  activities  include  establishing  key  process  objectives,  activities,  activity 
sequences  and  activity  products.  Then  using  the  operational  model  the  team  developed, 
basic  success  profiles  are  created  for  each  key  objective  as  shown  in  Figure  3. 
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Figure  3.  Basic  Success  Profiles  (Alberts,  et.  al.,  2008) 

These  represent  the  current  probability  or  likelihood  that  the  desired  outcome  will  be 

achieved  or  exceeded.  Then  a  causal  analysis  of  the  current  state  of  each  key  objective  is 
performed  by  determining  the  conditions  and  circumstances  that  either  positively  or 
negatively  affect  the  execution  of  the  distributed  process.  Next  the  uncertainty  ranges  are 
developed  for  each  key  objective  based  on  the  best-  and  worst-case  scenarios  and 


underlying  rationale/inherent  uncertainty  in  the  distributed  process  as  shown  in  Figure  4. 
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Figure  4.  Success  Profile  with  Uncertainty  Range  (Alberts,  et.  al.,  2008) 

Then  key  objective  event  sensitivity  and  causal  analysis  is  done  by  the  team  to  recognize 

the  conditions  and  circumstances  that  drive  the  potential  for  success  given  the  event’s 
occurrence  as  shown  in  Figure  5. 
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Figure  5.  Success  Profile  with  Event  Sensitivity  (Alberts,  et.  al.,  2008) 
Finally,  an  expanded  success  profile  as  seen  in  Figure  6  is  developed  from  the 

information  attained  for  each  key  objective  (Alberts,  et  al.,  2008). 
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Figure  6.  Expanded  Success  Profile  (Alberts,  et.  al.,  2008) 

The  individual  expanded  success  profiles  succinctly  portray  whether  gaps  or 

competencies  exist  between  the  desired  potential  for  success  of  the  objective  and  the 
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team’s  current  assessment  of  the  objective’s  potential  for  success.  Using  the  operational 
model  during  mission/event  analysis  improves  the  predictions  of  a  distributed  mission’s 
potential  for  success  over  traditional  risk-analysis’s  approach  of  using  tacit  understanding 
and  event  consequence  guesswork.  The  comprehensive  set  of  expanded  success  profiles 
provide  insight  into  the  issues  affecting  the  success  potential  of  the  process  and  a 
foundation  for  creating  detailed  improvement  strategies/plans  or  a  convincing  argument 
for  confidence  in  mission  success  (Alberts,  et  al.,  2008).  Now  that  a  MAAP  overview  is 
complete,  a  mechanism  to  support  and  understand  the  physical  architecture  facets 
included  in  the  expanded  success  profiles  is  needed.  A  discrete-event  simulation  fulfills 
this  need. 


2.3.2.  Discrete-Event  Simulation 

In  order  to  ensure  a  thorough  and  sound  study  was  conducted,  the  simulation 
study  steps  from  a  Discrete-Event  System  Simulation  by  Jerry  Banks  et.  al.  recommends 
the  following  protocol: 

Step  1 :  Problem  Formulation.  Establishes  a  statement  of  the  problem  to  be 
addressed  and  ensures  all  parties  understand  the  problem  being  analyzed  and 
agree  with  the  formulation  of  the  problem  statement. 

Step  2:  Setting  of  Objective  and  Overall  Project  Plan.  Identifies  the  questions  to 
be  answered  by  simulation  and  ensures  that  simulation  is  the  appropriate  method. 

Step  3:  Model  Conceptualization.  The  essential  features  of  the  problem  are 
abstracted  out  and  basic  assumptions  used  to  characterize  the  system  are  modified 
until  a  useful  approximation  results. 

Step  4:  Data  Collection.  The  compilation  of  all  the  input  data  required  to  drive  the 
model. 

Step  5:  Model  Translation.  Requires  the  conceptual  model  to  be  entered  into  a 
computational  or  computer-recognizable  format. 
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Step  6:  Verification.  Ensures  that  the  simulation  is  performing  properly. 

Step  7:  Validation.  Compares  the  model  to  actual  system  behavior. 

Step  8:  Experimental  Design.  Determines  any  alternatives  to  be  analyzed  with 
respect  to  any  factor  levels  to  include  replications,  run  time,  etc. 

Step  9:  Production  Runs  and  Analysis.  Estimate  the  system’s  measures  of 
performance  for  the  model  being  simulated. 

The  rest  of  this  research  paper  is  devoted  to  analyzing  the  TBMDS  with  the 
MOSAIC  management  approach  and  the  synergies  gained  by  incorporating  a  discrete 
event  simulation  and  the  MAAP  assessment  to  provide  mission  assurance  of  a  TBMDS. 
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Chapter  3.  Methodology  Under  MOSAIC  Approach 

The  MOSAIC  management  approach  was  used  to  analyze  a  TBMDS  with  two 
assessments:  a  discrete  event  simulation  and  MAAP.  The  discrete  event  simulation 
analysis  delves  into  the  physical  architecture  nuances  and  the  MAAP  assessment 
incorporates  the  discrete  event  simulation  analysis  with  all  the  other  factors  to  address  the 
system  risks  and  opportunities  at  a  consolidated  perspective. 

3.1.  Discrete  Event  Simulation 

The  purpose  of  the  discrete-event  simulation  analysis  was  two-fold:  1)  to 
prioritize  improvement  areas  of  a  TBMDS  by  looking  at  the  system  components  to 
determine  which  area(s)  of  improvement  have  the  biggest  impact  on  the  mission  and  2)  to 
better  understand  how  the  system  architecture  nuances  influence  the  mission  assurance  of 
the  TBMDS.  The  protocol  process  followed  below  is  limited  to  the  first  purpose  and  is 
not  to  be  confused  with  the  overall  TBMDS  mission  assurance  of  the  paper. 

3.1.1.  Step  1:  Problem  Formulation 

For  a  distributed  system  of  systems  like  the  TBMDS,  commanders  of  the 
individual  system  components  tend  to  advocate  for  improvements  in  their  own  area  of 
influence.  A  TBMDS  commander  needs  to  be  able  to  remove  the  bias  of  the  individual 
system  commanders  and  assess  the  performance  of  the  TBMDS  in  its  entirety.  The 
problem  addressed  by  the  discrete-event  simulation  is  the  TBMDS  commander’s 
requirement  to  maintain  or  enhance  the  mission  capability  of  the  distributed  TBMD 
system  of  systems.  For  this  study,  the  TBMDS  capability  assessment  ends  at  the  launch 
of  the  first  interceptor/shooter  in  response  to  a  ballistic  missile  threat. 
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3.1.2.  Step  2:  Setting  of  Objective  and  Overall  Project  Plan 

The  objective  of  the  discrete-event  simulation  consists  of  identifying  and 
prioritizing  the  components  of  the  TBMDS  that  impact  mission  success  the  most  as 
discussed  in  Chapter  1 .  The  overall  discrete-event  simulation  plan  is  to  run  a  Monte  Carlo 
simulation  of  the  model  developed  in  Arena  to  complete  the  objective. 

3.1.3.  Step  3:  Model  Conceptualization 

The  first  conceptual  model  was  portrayed  in  Chapter  2,  Figure  1  and  its  follow-on 
description.  Then  the  operational  sequence  model  from  the  MAAP  assessment  discussed 
in  section  3.2.  and  shown  in  Figure  1 1  provided  a  more  accurate  conceptual  model  with 
sequence  information  for  the  current  TBMDS  being  studied. 

3.1.4  Step  4:  Data  Collection 

At  a  high  level  of  abstraction,  the  only  data  required  to  model  the  TBMDS  is  the 
availability/reliability  for  each  component.  In  general,  the  best  source  for  this  data  is  their 
respective  historical  maintenance  database.  The  Mean  Time  to  Repair  (MTTR)  and  Mean 
Time  Between  Failures  (MTBF)  gathered  from  the  databases  provide  the  foundation  for 
calculating  the  component’s  availability/reliability.  Availability/reliability  determined  by 
this  method  calculates  the  unavailability  as  the  MTTR  divided  by  the  MTBF  and  thus  the 
system  availability/reliability  as  one  minus  the  unavailability  (Scheer  and  Moxley,  2005). 

The  Army  consolidates  this  information  for  their  radars  with  the  Army 
Maintenance  Database.  However,  EUCOM’s  AN/TPY-2  radar  is  currently  contractor 
maintained  and  therefore  the  historical  data  was  not  available.  Thus,  MTTR  and  MTBF 
were  obtained  from  the  Army  Maintenance  Database  for  Support  Equipment  for  a  similar 
radar  system  with  similar  complexity.  The  SMEs  assumed  that  systems  of  comparable 
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complexity  and  exposure  will  have  similar  failure  rates.  For  the  communication  links,  this 
process  was  already  conducted  in  a  digital  communication  reliability  paper  for  fiber  optic 
I/O  (Scheer  and  Moxley,  2005).  Therefore  the  availability/reliability  data  for  the 
communication  links  was  pulled  directly  from  the  paper.  The  interceptor  and  C2  facility 
availability  data  was  provided  by  subject  matter  experts  (SME). 

3.1.5.  Step  5:  Model  Translation 

The  Arena  simulation  software  was  used  to  help  demonstrate,  predict,  and 
measure  system  strategies  for  effective  and  efficient  performance.  The  Arena  simulation 
software  uses  representative  process  or  logic  modules  joined  by  connector  lines  to  show 
how  entities  flow  through  the  simulated  system  in  a  prescribed  way.  The  uniform 
distribution  and  Arena’s  default  stream  10  random  number  generator  sequence  was  used 
in  the  create  module,  called  Sensor  in  Figure  7,  to  simulate  arrival  of  theater  ballistic 
events.  In  addition,  the  following  assumptions  were  made: 

-  Only  1  event  enters  the  system  at  a  time 

-  Events  proceed  through  the  system  until  the  component  architecture  fails;  the 
model  does  not  take  into  account  any  early  termination  instances  other  than 
component  failure  since  such  circumstances  would  not  be  useful  for  reliability 
analysis  of  this  problem 

-  Reliability  and  availability  were  rolled  into  one  percentage  for  each  component 
since  events  proceed  through  the  system  so  quickly  and  infrequently  in  reality 

-  The  uniform  (0,  1)  distribution  may  not  be  the  appropriate  arrival  distribution; 
however  reliability  is  independent  of  the  arrival  distribution 
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Figure  7  depicts  the  Arena  model  of  the  system  from  the  identification  of  TBM  launch  by 
a  remote  sensor  to  interceptor/shooter  launch. 

3.1.6.  Step  6:  Verification 

A  number  of  procedures  were  used  to  verify  that  the  conceptual  model  was 
correctly  translated  into  a  simulation  model.  First,  the  number  of  entities  created  in  the 
model  was  compared  to  the  number  coming  out  of  the  system  to  ensure  they  were  equal. 
This  confirms  that  no  entities  were  being  stranded  anywhere  in  the  model.  The  next  step 
was  to  ensure  the  model  addresses  the  question  posed  so  entities  were  animated  and 
watched  as  they  flowed  through  the  model  to  ensure  they  ran  as  intended  and  were 
consistent  with  the  conceptual  model.  Finally,  the  data  was  checked  against  mathematic 
reliability  computations  which  showed  the  discrete  simulation  model  to  be  accurately 
depicting  the  system. 

3.1.7.  Step  7:  Validation 

Next,  the  model  was  validated  to  ensure  that  the  right  model  was  built.  The  first 
step  was  to  recognize  the  model’s  limitations  and  purpose.  The  purpose  of  the  model  was 
to  prioritize  the  components  used  to  complete  the  mission  for  the  launch  of  the  first 
interceptor/shooter  and  therefore  the  system  was  only  modeled  up  unto  that  point.  Since  a 
field  test  was  out  of  the  question,  the  results  were  assessed  and  no  significant  differences 
were  noted  between  the  model  and  reality. 

3.1.8.  Step  8:  Experimental  Design 

Following  the  Design  of  Experiment  process,  the  objective  was  set  to  prioritize 
the  top  components  that  impacted  mission  success  the  most.  Mission  success  in  this 
model  was  defined  as  the  launch  of  the  interceptor/shooter.  Normal  distributions  were 
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developed  around  the  appropriate  levels  for  all  the  components  to  set  the  range  of  factors 
to  be  analyzed.  The  normal  distributions  used  for  each  component  are  shown  in  Table  1. 
A  small  sample  size  was  analyzed  at  first,  and  then  the  appropriate  number  of  production 
events  was  established  at  100,000  events. 


Component 

Distribution  Parameters  (%) 

Radar 

Norm(97.9, 1) 

Comm 

Norm(99.9,  .3) 

C2  Fac 

Norm(99, 1) 

Shooter 

Norm(50, 10) 

Table  1.  Component  Reliability  Distribution 


3.1.9.  Step  9:  Production  Runs  and  Analysis 

Analysis  of  the  productions  runs  used  the  paired  t-test  confidence  interval  (Cl) 
with  error,  a,  equal  to  0.01  to  rank  each  scenario  with  results  discussed  in  Chapter  4.  A 
paired  t-test  looks  at  paired  sets  of  measured  values  and  determines  whether  they  differ 
from  each  other  in  a  significant  way.  A  graphical  representation  of  the  t-test  can  depict 
one  of  the  three  possible  results  as  shown  in  Figure  8  below. 


•  Cl  to  left  of  0:  then  02  >  0t 

0^4) 


i - X— ) - b 


(1-tf)  Cl 

■  Cl  to  right  of  0:  then  0,  <  Gj 


+-( — x — y 


o 


-  CI  includes  0:  then  cannot  reject  the  null 


(IX  ) 


Figure  8.  Graphical  Representation  of  paired  t-test  (Johnson,  2011) 
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3.2.  MAAP 


With  the  methodology  for  the  simulation  part  of  the  study  complete,  the 
remaining  section  of  this  chapter  describes  the  specifics  for  the  TBMDS  MAAP 
assessment. 

Since  this  is  a  feasibility  study,  Phase  1 ,  Preparing  for  the  Assessment,  and  Phase 
3,  Post  Assessment  Activities,  of  the  MAAP  were  unnecessary  and  therefore  not 
included.  Additionally,  at  a  high  level  of  abstraction  for  the  MAAP  assessment,  the 
mission  does  not  lend  itself  to  be  broken  down  to  only  consider  launch  of  the  interceptor 
and  therefore  for  the  MAAP  assessment  the  shooter  tracking  and  interception  was  also 
taken  into  consideration  to  assess  mission  of  the  entire  TBMDS.  This  required  the 
following  assumptions  to  be  made  for  the  MAAP  assessment: 

-  A  discrete  event  simulation  of  the  entire  TBMDS  would  have  similar  results  to 
the  simulation  performed  in  this  paper  with  the  additional  reliability  of 
interception  for  illustrative  purposes  being  50%  (Pena,  1998) 

-  Lower  level  key  objectives,  activities,  work  products,  etc  (not  in  the  scope  of 
this  paper)  support  the  roll-up  of  the  top  level  assessment 

Following  the  MAAP  activities  described  in  Chapter  2  Literature  Review,  the 
operational  model  was  developed  using  the  conceptual  model  from  Figure  1.  Four  key 
objectives  were  established  for  the  TBMDS:  1)  negate  effectiveness  of  TBM,  2)  negate 
likelihood  of  TBM,  3)  protect/defend  forces/population  centers,  and  4)  strengthening  US 
security  relationships  with  allies  (Gompert  et.  al.,  1999;  Wilkening,  1999;  SECDEF, 
1995).  Specifically  the  negate  effectiveness  key  objective  was  understood  to  include  not 
only  the  personnel  aspect  of  Key  Objective  3  but  also  the  effect  of  TBM  to  property, 
possessions  and  livelihood.  In  addition,  Key  Objective  2  was  understood  to  address  the 
deterrence  aspect  of  the  TBMDS.  The  following  key  activities  were  derived  from  the 
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researcher’s  analysis  of  the  operational  model  to  support  the  key  objectives:  warn 
coalition  partner,  order  response  to  TBM,  track  targets,  communicate  target  tracks, 
identify  hostile  threats  to  the  theater,  locate  hostile  TBM  and  launch  interceptor.  For  the 
TBMDS,  the  work  products  and  key  activities  were  synonymous  and  therefore  a 
distinction  between  the  two  was  not  used  for  this  assessment.  At  this  high  level  of 
abstraction,  all  the  key  objectives  and  activities  were  able  to  be  aligned  with  higher 
headquarters’  mission,  the  Chairman  of  the  Joint  Chiefs  of  Staff  Universal  Joint  Task  List 
(CJCS,  2002),  and  therefore  were  candidates  for  further  mission  assurance  analysis.  The 
analysis  presented  here  is  offered  primarily  to  demonstrate  how  the  MAAP  process  could 
be  used  for  TBMDS  mission  assurance  modeling;  a  more  rigorous  analysis  with 
appropriate  SMEs  would  provide  more  useful  results.  For  each  key  objective,  basic 
success  profiles  were  built  by  assessing  the  lowest  level  of  success  tolerable  and  setting 
that  as  the  success  threshold.  For  Key  Objectives  1  and  3,  their  success  thresholds  were 
set  to  excellent  as  the  loss  of  any  life/livelihood  are  generally  unacceptable  from  the  US 
perspective.  Key  Objectives  2  and  4  were  set  to  high  since  success  in  these  areas  are 
highly  sought  after  but  have  a  lower  pain  threshold  and  therefore  a  lower  success 
tolerance  threshold.  To  assess  each  key  objective’s  current  potential  for  success,  analysis 
incorporated  not  only  the  full  TBMDS  discrete  event  simulation  analysis  and  the 
operational  model  which  essentially  represent  similar  TBMDS  information  but  other 
measures  are  that  supported  the  key  objective.  For  Key  Objective  1,  this  included 
hardening  capabilities  and  service/support  redundancies.  For  Key  Objective  2,  this 
included  political/economic  actions.  For  Key  Objective  3,  this  included  distance 
variability  from  interceptor  assets  and  hardening  capabilities.  For  Key  Objective  4,  this 
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included  memorandum  of  agreements  and  conducted  exercises.  The  actual  assessment  of 
the  top  level  key  objectives’  current  potential  for  success  is  notional  without  the 
underlying  roll-up  support  but  with  the  assumed  roll-up  support,  it  can  be  representative. 

Next,  the  success  profdes  with  uncertainty  ranges  were  created  by  adding  a  best- 
case  and  worst-case  potential  for  success  to  the  basic  success  profile.  All  the  same 
aspects  analyzed  for  the  basic  success  profiles  of  each  key  objective  were  reassessed  for 
their  best-case  and  worst-case  scenarios.  Finally  the  expanded  success  profiles  were 
created  for  each  key  objective  by  adding  the  event  sensitivity  analysis  to  the  success 
profiles  with  uncertainty  ranges.  In  keeping  with  the  intent  of  this  study  to  determine  the 
feasibleness  of  the  MAAP  assessment,  only  one  event  was  analyzed  for  all  four  key 
objectives,  loss  of  communications.  Once  again  after  a  loss  of  communications,  all  the 
same  aspects  analyzed  for  the  basic  success  profile  were  assessed  and  brought  the 
potential  for  success  of  Key  Objectives  1,  2,  and  4  to  low  and  for  Key  Objective  3  to 
minimal.  The  results  and  analysis  of  the  MAAP  assessment  is  the  next  step  in  the  process 
and  leads  us  into  Chapter  4  Results  and  Analysis. 


28 


Chapter  4.  Results  and  Analysis 

Though  the  discrete  event  simulation  and  MAAP  assessments  have  essentially 


been  discussed  separately,  the  two  assessments  were  interwoven  and  supported  each 
other.  For  example,  results  from  one  assessment  method  were  used  to  support  the  other 
method,  and  vice  versa.  However,  to  simplify  the  discussion,  the  discrete  event 
simulation  results  and  analysis  will  be  discussed  first,  followed  by  the  analysis  for 
MAAP’s  expanded  success  profiles  and  finally  the  key  findings  of  the  combination  of  the 
two  analyses  will  be  reviewed. 

4.1.  Discrete  Event  Simulation  Results 

At  the  level  of  analysis  conducted  by  this  study,  the  discrete  event  simulation 
model  resulted  in  a  series  system.  At  first  glance,  this  appeared  to  be  a  wonderfully 
simplifying  conclusion  since  reliability  importance  theory  shows  “that  the  least  reliable 
component  in  a  series  system  has  the  biggest  effect  on  the  system  reliability”  (ReliaSoft 
Corporation,  2007).  Therefore,  improvement  to  the  component  with  the  least  reliability 
will  always  have  the  biggest  impact  on  the  mission  from  a  system  reliability  perspective. 
However,  a  TBMDS  while  still  a  series  system  uses  some  components  more  often  than 
others  and  therefore  a  discrete  event  simulation  was  able  to  resolve  the  component 
utilization  discrepancy  not  as  easily  computed  by  reliability  importance  theory.  Using  the 
reliability  distributions  in  Table  1,  the  executed  discrete  event  simulation  of  the  base 
system  is  shown  in  Figure  9.  Figure  9  shows  the  results  of  one  production  run  of  the  Base 
system  annotated  with  the  count  of  entities  for  each  path.  Mission  success  in  terms  of  the 
launch  of  an  interceptor/shooter  was  47586  beside  the  via  shooter  module  for  this  run. 
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In  order  to  understand  the  significance  of  component  improvements,  three 
alternatives  to  the  base  system  were  initially  analyzed.  Alternative  1  increased  the  shooter 
availability/reliability  from  a  mean  of  50  to  a  normal  distribution  with  a  mean  of  75  and 
variance  of  10  from  Table  1.  Alternative  2  increased  the  C2  Facility 
availability/reliability  to  a  normal  distribution  with  a  mean  of  99.9.  Alternative  3 
increased  the  Radar  availability/reliability  to  a  normal  distribution  with  a  mean  of  99.  For 
each  of  these  alternatives  the  discrete  event  simulation  model  in  Figure  9  was  run  again. 
Figure  10  shows  the  resulting  paired  t-test  comparison  of  the  combination  of  the  Base 
system,  denoted  as  Base,  and  the  three  alternatives,  designated  by  their  number,  taken  in 
pairs. 
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Figure  10.  Mission  Success  Paired  t-test 

Figure  10  shows  a  statistically  significant  improvement  for  Alternative  1  indicating  that 
an  improvement  in  the  shooter  availability  is  the  best  alternative  system.  To  bring  fidelity 
to  the  remaining  component  improvement  alternatives,  a  second  paired  t-test  was 
conducted  with  mission  success  modified  to  the  successful  notification  of  the  coalition 
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partner.  The  Alternatives  remained  the  same  except  Alternative  1  was  modified  to  an 
improvement  of  the  communication  component  with  a  normal  distribution  mean  of  99.99. 
Figure  1 1  shows  the  resulting  graphical  depiction.  Though  all  the  alternatives  have 
statistically  significant  improvements  over  the  base  system,  the  margin  of  improvement  is 
relatively  small  compared  to  the  system  reliability  improvement  to  be  gained  by  the 
shooter  component. 


Figure  11.  CP  notification  Paired  t-test 

Examining  the  paired  t-test  results  in  Figures  10-11,  resulted  in  the  component 
prioritization  listed  in  Table  2. 


Component 

Improvement 
Mission  Impact 

Shooter 

Most 

C2  Fac 

ft 

Radar 

¥ 

Comm 

Least 

Table  2.  Component  Prioritization 
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Table  2  does  not  take  into  account  any  cost  benefit  analysis,  but  from  a  system  reliability 
perspective  it  prioritizes  the  component  improvements  that  have  the  biggest  impact  on  the 
mission. 

4.2.  MAAP  Success  Profiles 

As  experience  has  taught  us,  system  reliability  alone  does  not  indicate  mission 
assurance.  Though  the  component/system  reliability  does  play  a  part  and  therefore  was 
incorporated  into  the  MAAP  assessment3.  After  reviewing  the  conceptual  model  in 
Figure  1  and  the  figure’s  description,  the  detailed  operational  model  in  Figure  12  was 
developed  to  ground  the  remaining  MAAP  analysis. 

Then  the  key  objectives  and  activities  were  established  and  aligned  with  the 
Chairman  of  the  Joint  Chiefs  of  Staff  Universal  Joint  Task  List  (CJCS,  2002).  Key 
objectives  that  would  not  have  aligned  with  direction  from  higher  headquarters  would 
have  been  an  indicator  of  potential  improvement  areas  or  candidates  for  re-alignment. 
However,  the  four  key  objectives  established  for  the  TBMDS,  readily  aligned  to  the 
Chairman  of  the  Joint  Chiefs  of  Staff  Universal  Joint  Task  List  (CJCS,  2002)  as  shown  in 
Figure  13. 


For  this  feasibility  study,  the  author  instead  of  a  team  of  experts  completed  the  MAAP 
assessment. 
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Figure  13.  Key  Objective  and  Activities 

Armed  with  the  operational  model  from  Figure  12  and  the  establishment  of  the 
key  objectives  and  activities  in  Figure  13,  the  analysis  of  the  TBMDS’s  potential  for 
mission  success  evolved  from  basic  success  profiles  through  the  steps  described  in 
section  2.3.1  and  explained  in  section  3.2  to  the  expanded  success  profiles  shown  in 
Figures  14-17.  As  shown,  Key  Objectives  1,  2  &  3  of  the  TBMDS  have  gaps  between  the 
current  potential  for  success  and  the  desired  success  thresholds.  However,  Key  Objective 
2  has  an  uncertainty  range  that  has  the  potential  to  eliminate  its  gap.  Unfortunately,  all 
four  key  objectives  are  still  sensitive  to  event  occurrences  that  degrade  their  potential  for 


35 


success  to  low  or  minimal  levels.  Therefore  the  analyzed  TBMDS  has  many  areas  of 
improvement  to  reach  the  desired  success  threshold. 
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Figure  14.  Key  Objective  1 
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Figure  15.  Key  Objective  2 
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Figure  16.  Key  Objective  3 
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4.3.  Summary  of  Analysis  Results 

The  discrete  event  simulation  analysis  provided  the  component  improvement 
prioritization  of  shooter/inceptor,  C2  Facility,  Radar  and  Communications.  The  MAAP 
identified  gaps  between  the  current  potential  for  success  and  the  desired  success 
thresholds  for  three  of  the  four  key  objectives.  Both  analyses  indicate  that  the  TBMDS 
has  warranted  improvements  to  be  made.  The  MOSAIC  management  paradigm  from 
Figure  2  would  use  the  details  provided  by  the  discrete  event  simulation  and  MAAP 
assessments  to  develop  an  action  plan  for  improving  the  TBMDS’s  potential  for  success 
in  this  case.  Then  commanders  would  follow  though  by  implementing,  tracking  and 
controlling  the  planned  improvements. 
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Chapter  5.  Discussion  and  Conclusions 

With  the  financial  constraints  faced  by  commanders  today  and  the  oversight 
technology  immediately  provides  when  mission  success  is  in  question,  commanders  need 


the  efficient  mission  assurance  management  approach  MOSAIC  can  provide.  From 
prioritizing  and  justifying  funding  requests  to  operational  mission  assurance,  MOSAIC 
has  the  modularity  to  handle  the  range  of  complex  systems.  The  TBMDS  mission 
assurance  approach  captured  in  this  research  represents  a  methodology  field  commanders 
can  utilize  by  the  very  nature  of  having  a  command,  the  mission,  and  furthermore,  the 
alignment  of  their  mission  to  the  next  higher  level  of  command’s  mission  described  in 
this  paper  as  the  enterprise-wise  alignment  of  key  objectives.  Additionally,  the  MAAP 
assessment  has  the  comprehensive  analysis  required  to  capture  the  complex 
environmental  interactions  inherently  found  in  a  TBMDS. 

This  research  focused  mainly  on  the  assessment  activity  within  MOSAIC  to  show 
an  alternative  to  the  traditional  risk  management  approach.  By  incorporating  the 
assessments  of  SEI  MAAP  and  a  standard  discrete  event  simulation,  not  only  were 
synergies  recognized  within  the  processes  but  also  a  balance  between  subjectivity  and 
objectivity  was  achieved.  The  assessment  piece  provides  a  solid  foundation  for  the 
follow-on  activities  of  planning,  implementing,  tracking  and  controlling  necessary  to 
achieve/maintain  the  desired  outcome.  The  follow-on  activities  are  generally  better 
understood  and  incorporated  and  therefore  were  included  more  for  completeness  of  the 
MOSAIC  approach  than  as  additional  areas  of  research. 

From  a  high  level  of  abstraction,  MOSAIC’S  discrete  event  simulation  and  MAAP 
assessments  indicate  that  the  TBMDS  has  potential  improvement  areas  before  mission 
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assurance  is  established.  However,  further  research  in  this  area  could  use  a  specific 
system’s  actual  component  reliabilities  and  delve  into  the  enterprise-wide  alignment  of 
key  objectives  down  to  the  operational  processes  which  include  all  tasks,  policies, 
procedures,  organizations,  people,  technologies,  tools,  data,  inputs,  and  outputs  needed  to 
achieve  the  specified  mission  using  an  team  of  experts  for  an  in  depth  MAAP  assessment. 
Additionally,  this  study  only  took  into  consideration  the  components  of  the  US  TBMDS 
system  from  a  EUCOM  perspective.  As  agreements  are  put  into  place  and  theaters  begin 
to  rely  on  the  assets  of  multiple  nations,  a  combined  organization  like  NATO,  could  use 
the  assurance  a  theater-wide  TBMDS  MOSAIC  approach  would  provide. 
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